Playwright fix test failures and improve the pytest_runtest_makereport hook #6599
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes several changes to improve error handling, and debugging capabilities.
The most important changes include handling the 502 errors for the test_loginless_mozilla_account_aaq test (which was the main flakiness reason for that test), extending timeouts for two messaging system tests (Sometimes SuMo returns the expected search results with a >10 seconds delay so I've bumped the wait to 15) & skipped the AAQ quote test for now .
Error Handling Improvements:
playwright_tests/tests/ask_a_question_tests/aaq_tests/test_aaq_form_page.py
: Added a check to skip the sign-in button click if a 502 error occurs and the auth page is already displayed due to a forced page refresh.playwright_tests/tests/conftest.py
: Refactored thepytest_runtest_makereport
hook to ensure video recordings are properly saved and attached to the Allure report when a test fails by taking into account not only the call.excinfo (exception is raised) but also the pytest-check failures.Test Reliability Enhancements:
playwright_tests/tests/ask_a_question_tests/aaq_tests/test_posted_questions.py
: Skipped thetest_quote_reply_functionality
test until a decision is made on whether to revert a related change.playwright_tests/tests/explore_help_articles_tests/explore_by_topic_tests/test_explore_by_topics.py
: Added navigation expectation and refresh logic to handle cases where navigation does not occur => causes the test to fail.Timeout Adjustments:
playwright_tests/tests/messaging_system_tests/test_messaging_system.py
: Increased the timeout for verifying that no users are returned inside the messaging system search results in two separate tests. [1] [2]